Companies today are tasked with implementing solutions for many types of integration challenges within their respective enterprises. Many of these challenges involve issues of application integration (e.g., integration among and/or between software applications and/or other systems) and fall into common patterns. For instance, globalization-related trends have prompted business houses stationed across the geographies to participate in active trading worldwide. But these new opportunities come with the challenge of integration. Whether business-to-business (B2B) transactions across borders will be successful can depend on numerous factors such as, for example, the ease of adoption of new technologies, lighter on-premises installations, scalability (e.g., to address spikes in transaction volumes), improved or optimized bandwidth utilization, maintaining the proper ordering of transactions, etc. Many B2B transactions currently are facilitated by exchanging business payloads over Internet.
For example, a first challenge area relates to propagation of similar business objects from one system to multiple other systems, such as, for example, in an order status change or a product price change. A second area relates to synchronization of similar business objects between two or more systems to obtain a single view, such as, for example, a real-time synchronization of customer, product registration, product order, and product SKU information among several applications. This is one of the most common issues requiring an integration solution. In a one-way synchronization, there generally is one system (e.g., resource) that acts as a data source and one or more resources that are targets of the synchronization. In a two-way synchronization, every resource often is both a potential source and target of a synchronization. There generally is not a single resource that acts as the primary data resource. Thus, a change to any resource should be reflected in all other resources. A third area involves information joined from multiple sources into a common destination system, such as, for example, communicating pharmacy customer records and prescription transactions and website data into a central application and database.
Various tools have been provided that enable a user to design and deploy solutions that address these challenges using, for example, the publish-and-subscribe model or one of its variants. The publish-and-subscribe model is a specific type of message-based solution in which messages are exchanged anonymously through a message broker. Applications that produce information that needs to be shared make this information available in specific types of recognizable documents that they publish to the message broker. Applications that require information subscribe to the document types they need.
At runtime, the message broker receives documents from publishers and then distributes the documents to subscribers. The subscribing application processes or performs work using the document and may or may not send a response to the publishing application.
In a typical system, an integration server or applications running on an integration server publish documents to a broker. The broker then routes the documents to subscribers located on other integration servers. The integration server and the broker share a fast, efficient process for exchanging documents across the entire system.
Brokers can be linked to form units known as territories, and brokers that form a territory may sometimes function as a single logical broker. Clients that connect to one broker in a territory can automatically exchange documents with clients connected to any of the brokers in the territory. Territories can be organized in any desired way, e.g., to take into account factors such as, for example, geographical distribution of clients and/or back-end services, the types of services offered, actual and/or expected traffic, actual and/or expected processing loads, etc. Territories thus may help support scalability by allowing one to segment traffic among multiple brokers, while still treating the brokers as a single operational unit.
Alternatively, or in addition, brokers can be grouped into clusters. As used herein and is understood by those skilled in the art, territories and clusters are different concepts. Clusters may be thought of as including a set of loosely or tightly connected computers (depending on the implementation) that work together so that they can in many respects be viewed as a single system. Territories as described herein typically will have no master/slave relationship, whereas the same is not necessarily true when clusters are involved. In this vein, one further difference as between territories and clusters may relate to the way in which clusters oftentimes are set up and run (e.g., apart from the master/slave concept) and, more particularly, to the notion of typically using clusters to distribute workload and solve performance and/or reliability issues with a system, as these concepts generally are not associated with what a territory does. For instance, a territory may use a set of brokers to distribute the workload, and also frequently may involve a message, document, or the like, being published to all subscribers across all brokers at the same time, e.g., such that a publisher client only has to publish to a nearest (or any) broker, with the system ensuring that subscriber clients in the group (allowed to receive the publication) can access it from its nearest (or any) broker in the territory (or even through a territory gateway to another territory). By contrast, a cluster might in some cases help to distribute the workload amongst the brokers in a cluster, but involve a message, document, or the like, being published to only one broker in a cluster.
Consortiums of companies in and across industries have come forward in attempts to standardize the payload format for which these and/or other transactions are made. Yet traditional solutions of processing B2B transactions directly between senders and receivers or via a B2B exchange continue to face challenges related to the sending/receiving of large business payloads, as these are bandwidth and other resource intensive operations. In addition, the ability to dynamically scale up on-demand while still maintain the order of processing the payloads with neither the sending party nor the receiving party becoming a bottleneck of the transaction can be a challenge.
That is, because current technology current solutions process B2B transactions directly between the sender and receiver, or via a B2B exchange, optimizations in processing large payloads and in turn maintaining the order in which these payloads are processed typically is performed via the sender/receiver of the B2B transaction. Thus, at least one of the entities needs to have processing capabilities, and the overall data footprint over the network can be high. In addition, both sender and receiver side clients need to be up to ensure that the payload is properly sent and received. This can be particularly challenging when so-called small and medium enterprises (SMEs) do not have a continuous Internet presence.
Thus, it will be appreciated that there is a need in the art for improved integration techniques, e.g., in situations where large amounts of data are to be processed, the ordering of the processing is to be maintained, etc.
One aspect of certain example embodiments relates to a framework that helps address dynamic scalability and in-order processing of payloads, e.g., pertaining to B2B and/or other transactions.
Another aspect of certain example embodiments relates to enabling a receiver and/or sender to dynamically and/or statically provide an executable to an intermediary entity (referred to as a central payload processor (CPP) in certain example embodiments) to process a payload, en-route. This arrangement advantageously provides a very efficient way of delivering ordered content, especially when the content is extremely large.
Still another aspect of certain example embodiments relates to leveraging cloud computing techniques to make available high computing power and geographical distribution of storage space (e.g., even to SMEs).
Still another aspect of certain example embodiments relates to enabling the receiver to be down when the sender sends the payload, while still maintaining the order of the payload, and potentially enabling the payload to be processed and the results delivered later.
Yet another aspect of certain example embodiments relates to implementing map-reduce functionality at the intermediary entity that has high processing capabilities.
Certain example embodiments make use a CPP as a pre-agreed upon unit provided between senders and receivers, where the sender of a B2B payload does not send the payload directly to the receiver. It instead sends the payload to the CPP, and the CPP includes processing resources sufficient to use preconfigured executable programs and/or fetch executable programs dynamically from the receiver, e.g., to process the payload. The CPP thus is able to handle large payloads and maintain the order of the processing. This approach is advantageous in that the receiver need not have high processing capabilities, the sending entity can have payloads intended to be sent to multiple receiving entities being processed with a much smaller network data footprint, receivers need not be up when the sender sends the payload, etc.
In certain example embodiments, there is provided a method of processing at a CPP payloads sent from a sending entity (e.g., on behalf of one or more receiving entities). The method comprises: receiving, from the sending entity (e.g., and at the CPP), a payload to be processed for the one or more receiving entities; and determining, for each of the one or more receiving entities, whether an appropriate bundle of executable logic is currently available at the CPP for processing the received payload for the respective receiving entity. For each said receiving entity: in response to a determination that there is an appropriate bundle of executable logic currently available at the CPP for processing the received payload for the respective receiving entity, the received payload is processed using at least one processor of the CPP in connection with that currently available bundle of executable logic; and in response to a determination that there are no appropriate bundles of executable logic currently available at the CPP for processing the received payload for the respective receiving entity, an attempt is made to retrieve a new bundle of executable logic from the one or more receiving entities. Following successful processing of the received payload, results of the successful processing are transmitted to the one or more receiving entities and/or stored in a non-transitory computer readable storage medium of the CPP.
According to certain example embodiments, the payload to be processed may be received from the sending entity in pre-formed fragments generated at the sending entity.
According to certain example embodiments, the method may further include receiving, from the sending entity (e.g., and at the CPP), a group of payloads to be processed together for the one or more receiving entities. Moreover, an indication of an order in which the payloads in the group are to be processed may be received, and the payloads in the group may be processed in that order. This may be accomplished in certain example embodiments by synchronizing a payload registry of the CPP with a payload registry of the sending entity in order to ensure that the order is maintained.
According to certain example embodiments, the method further may further involve waiting to transmit to the one or more receiving entities successful processing results, e.g., until all payloads in the group are successfully processed.
According to certain example embodiments, when an attempt to retrieve a new bundle of executable logic from the one or more receiving entities fails, the payload may be maintained at the CPP until a subsequent attempt to retrieve a new bundle of executable logic from the one or more receiving entities is successful; and when an attempt to retrieve a new bundle of executable logic from the one or more receiving entities is successful, the received payload may be processed using at least one processor of the CPP in connection with that newly retrieved bundle of executable logic.
In certain example embodiments, a central payload processor is provided, and it comprises: at least one processor and a memory; a file system backed by non-transitory computer readable storage media; a cache of executable programs; and a communication channel configured to receive, from a sending entity, a group of payloads to be processed for at least one receiving entity. In response to a group of payloads being received over the communication channel, the at least one processor, for each said payload in the group, is configured to: determine, for each said receiving entity, whether an executable program is currently available in the cache for processing the respective payload for the respective receiving entity; for each said receiving entity: (a) in response to a determination that there is an executable program currently available in the cache for processing the respective payload for the respective receiving entity, process the respective payload using the at least one processor of the CPP in connection with that currently available executable program, and (b) in response to a determination that there are no executable programs currently available in the cache for processing the respective payload for the respective receiving entity, retrieve a new executable program from the respective receiving entity and process the respective payload using the at least one processor of the CPP in connection with the newly retrieved executable program; and following successful processing of the respective payload, transmit results of the successful processing to respective receiving entity.
In certain example embodiments, a computer system is provided and it includes at least one sending entity and at least one receiving entity. The computer system also comprises a payload processor including: at least one processor and a memory, a file system backed by non-transitory computer readable storage media, a cache of executable programs, and a communication channel configured to receive, from the at least one sending entity, a group of payloads to be processed for at least one receiving entity. In response to a group of payloads being received over the communication channel, the at least one processor of the payload processor, for each said payload in the group, is configured to: determine, for each said receiving entity, whether an executable program is currently available in the cache for processing the respective payload for the respective receiving entity; for each said receiving entity: (a) in response to a determination that there is an executable program currently available in the cache for processing the respective payload for the respective receiving entity, process the respective payload using the at least one processor of the payload processor in connection with that currently available executable program, and (b) in response to a determination that there are no executable programs currently available in the cache for processing the respective payload for the respective receiving entity, retrieve a new executable program from the respective receiving entity and process the respective payload using the at least one processor of the payload processor in connection with the newly retrieved executable program; and following successful processing of the respective payload, transmit results of the successful processing to respective receiving entity.
In certain example embodiments, there is provided a non-transitory computer readable storage medium tangibly storing instructions that, when executed by at least one processor of a system, perform the above-described and/or other methods.
These aspects and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments of this invention.